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The  air  safety  record  has  improved  substantially  in 
the  decades  since  introduction  of  the  jet  transport 
in  1959  and  throughout  the  period  crew  caused 
accidents  dominated  all  other  accident  causes  by  a  wide 
margin  (Figure  1).  It  has  consistently  accounted  for  over 
70%  of  the  major  (fatal  or  hull  loss)  accidents  in  spite  of 
increasing  attention  to  cockpit  resource  management 
programs  and  other  human  factors  areas. 


In  1986  we  concluded  a  study  of  accident  reports  to  better 
understand  accident  cause  factors.  A  total  of  126  major 
accidents  which  occurred  worldwide  during  the  period 
1977  through  1984  were  examined.  From  this  group  of 
accidents,  93  were  identified  where  enough  information 
was  available  to  allow  the  determination  of  significant 
cause  factors.  The  crew  cause  factor  classifications  and 
their  percentage  of  presence  in  these  93  accidents  are 


PRIMARY  CAUSE  FACTORS  -  HULL  LOSS  ACCIDENTS* 
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Figure  1.  The  distribution  of  causal  factors  in  hull  loss  accidents 
between  1959-1985  as  compared  to  1976-1985  is  depicted  here. 
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shown  in  Table  1.  Effective  control  of  these  flight  crew 
accident  factors  will  depend  largely  on  the  airline 
operation  and  training  community. 


33%  Pilot  deviated  from  basic  operational  procedures. 

26%  Inadequate  crosscheck  by  2nd  crew  member. 

99c  Crews  not  conditioned  for  proper  response  during 
abnormal  conditions. 

6%  Pilot  did  not  recognize  the  need  for  go-around. 

4 9c  Pilot  incapacitation. 

49c  Inadequate  piloting  skills. 

39c  Pilot  used  improper  procedure  during  go-around. 

39c  Crew  errors  during  training  flights. 

39c  Pilot  not  trained  to  respond  promptly  to  GPWS 
command. 

3 9C  Pilot  unable  to  execute  safe  landing  or  go-around 
when  runway  sighting  is  lost  below  MDA  or  DH. 

39c  Operational  procedures  did  not  require  use  of 
available  approach  aids. 

3 9c  Captain  inexperienced  in  aircraft  type. 

Table  I.  Significant  crew  cause  factors  and 

percentage  of  presence  in  93  major  accidents. 


Examination  of  worldwide  accident  history  shows  that 
some  operators  have  more  accidents  than  others.  Using 
the  Boeing  fleet  over  a  ten-year  period  as  an  example, 
16%  of  the  operators  have  crew  caused  accident  rates 
higher  than  the  fleet  average  and  these  operators  account 
for  over  80%  of  the  total  accidents.  Conversely,  80%  of 
the  operators  had  no  crew  caused  accidents  over  the  same 
period.  The  distribution  of  accidents  for  the  fleet  is  shown 
in  Figure  2. 
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Figure  2.  The  distribution  of  crew  caused  accidents  involv¬ 
ing  operators  of  Boeing  airplanes  between  1975-1984. 


Boeing  recently  contacted  a  small  group  of  operators,  most 
of  which  had  better  than  average  crew  caused  accident 
history.  This  was  done  with  a  view  to  obtaining 
information  on  techniques  which  contribute  to  safe 
operations  and  which  could  be  brought  to  the  attention 
of  all  operators  through  our  Flight  Operations  Support 
Program  (FOSP)  where  we  provide  pilot-to-pilot  contact 
with  the  operators.  Over  a  two-year  period  12  operators 
were  contacted.  Meetings  with  management  pilots  in  the 
flight  operations  and  training  departments  focused  on 
what  was  considered  by  the  operators  to  be  key  elements 
of  safe  operation.  It  is  the  results  of  these  informal 
meetings  that  are  summarized  here.  To  avoid  repetitious 
qualifying  phrases  denoting  the  degree  of  consensus  on  any 
point  the  reader  may  assume  a  reasonable  consensus  unless 
the  individuality  of  the  comment  is  emphasized.  The 
findings  are  grouped  for  convenience  of  presentation  only. 
The  format  does  not  imply  a  structured  approach  to  the 
discussions  held  with  the  operators. 

Management  Emphasis  on  Safety 

In  the  broad  context  of  management  these  operators 
characterize  safety  as  beginning  at  the  top  of  the 
organization  with  a  strong  emphasis  on  safety  and  this 
permeates  the  entire  operation.  Flight  operations  and 
training  managers  recognize  their  responsibility  to  flight 
safety  and  are  dedicated  to  creating  and  enforcing  safety- 
oriented  policies.  The  presence  or  absence  of  a  safety 
organization  did  not  alter  the  total  involvement  of  these 
managers.  However,  a  majority  of  the  operators  did 
maintain  an  identifiable  flight  safety  focal  point.  There  is 
an  acute  awareness  of  the  factors  that  result  in  accidents, 
and  management  reviews  accidents  and  incidents  in  their 
own  airline  and  in  other  airlines  and  alters  their  policies 
and  procedures  to  best  guard  against  recurrence.  There 
is  a  method  for  getting  information  to  the  flight  crews 
expeditiously  and  a  policy  that  encourages  confidential 
feedback  from  pilots  to  management.  This  management 
attitude,  while  somewhat  difficult  to  describe,  is  a  dynamic 
force  that  sets  the  stage  for  standardization  and  discipline 
in  the  cockpit  brought  about  and  reinforced  by  a  training 
program  oriented  to  safety  issues. 

Standardization  and  Discipline 

Management  recognizes  the  need  for  aircrews  performing 
in  a  standardized  way  and  the  importance  of  cockpit 
discipline  in  providing  the  environment  for  proper  crew 
coordination.  This  results  in  a  number  of  specific  policy 
implementations: 

•  A  standardization  pilot  and  a  standardization  flight 
engineer  (if  applicable)  are  appointed,  usually  one  for 
each  type  airplane  in  the  fleet.  All  procedure  and 
checklist  issues  are  coordinated  by  the  standardization 
personnel  recognizing  differences  in  airplane  and 
cockpit  configuration  within  the  type. 
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<»  A  strong  check  airman  program  acts  as  a  continuous 
quality  control  check  on  the  training  department. 
Standards  for  check  airman  candidates  exist  in  writing 
and  the  highest  level  of  flight  operations  management 
participates  in  the  evaluation  and  selection  process. 
Methods  exist  for  assuring  the  uniformity  of  check 
pilot  techniques  and  instruction,  usually  accomplished 
during  periodic  (monthly)  meetings  of  all  check  pilots. 
There  is  a  special  system  of  recurrent  checks  for  check 
pilots  that  is  indepenuent  of  the  line  pilot  recurrent 
training  program.  An  effort  is  made  to  assure  the 
uniformity  of  checking  techniques  by  correlating 
reported  non-standard  behavior  in  students  to 
individual  check  pilots  where  possible. 


•  There  is  a  firm  requirement  for  in-depth  takeoff  and 
approach  briefings  for  each  flight  segment.  This 
provides  the  entire  crew  with  knowledge  of  precisely 
how  the  event  is  to  be  performed.  The  takeoff  briefing 
is  a  review  of  departure  plans,  rejected  takeoff  (RTO) 
technique,  and  contingency  plans  in  the  event  of  an 
emergency  calling  for  return  to  the  departure  airport. 
One  operator  requires  an  RTO  touch  drill  where  each 
control  used  during  RTO  is  sequentially  touched  by 
the  pilot  who  will  be  making  the  takeoff. 


The  approach  briefing  is  usually  done  at  the  top  of 
descent  before  workload  increases.  It  covers  the 
navigation,  communication  and  procedural  details  of 
the  approach  for  the  specific  runway  involved, 
including  missed  approach  details. 


Cockpit  procedural  language  is  tightly  controlled  to 
maintain  consistency  and  to  avoid  confusion  from 
non-standard  callouts  which  can  result  from  crew 
members  using  differing  phraseology.  Callouts  and 
responses  are  done  verbatim.  The  recurrent  training 
program  and  check  pilot  system  rigidly  enforce  this 
requirement. 


•  The  large  operators,  operating  several  airplane  types 
(often  of  more  than  one  manufacturer)  devote  a  good 
deal  of  effort  to  standardize  checklists  and  operations 
manuals  to  meet  their  operational  philosophy  and 
format.  A  few  f 'el  that  cockpit  configuration 
standardization  is  essential  and  that  the  money  is  well 
spent  to  keep  cockpits  as  nearly  alike  as  possible. 


All  operators  surveyed  prefer  accomplishing  their  own 
training  so  that  positive  control  of  standardization  and 
discipline  can  be  maintained.  About  half  provide 
training  for  other  operators.  They  voiced  concern  that 
in  order  to  cut  costs,  buyers  of  training  will  request 
a  training  program  less  extensive  than  the  seller  uses 
to  train  his  own  crews  and  noted  the  difficulties  this 
imposed  on  training  contract  negotiations.  Of  course 
where  the  seller  leases  unmanned  facilities  to  a  buyer 


AIRLINER/APR-JUN  1987 


who  provides  the  training  personnel  the  seller  loses  any 
say  in  the  quality  of  training. 


It  was  also  felt  that  airlines  that  buy  training  from  the 
lowest  bidder  may  experience  a  lack  of  standardiza¬ 
tion.  This  results  from  the  fact  that:  1)  New-hire  rated 
pilots  may  not  receive  complete  coverage  of  company 
philosophy  and  procedures,  2)  Initial  and  recurrent 
training  may  not  be  bought  from  the  same  source  and 
3)  The  source  of  recurrent  training  may  change  from 
time  to  time.  Too  often  the  airlines  we  visited  who 
perform  contract  training  see  a  crew  arrive  for  training 
where  each  pilot’s  prior  training  has  been  with  a 
different  contractor.  Sometimes  the  pilots  even  have 
different  operating  manuals  which  were  obtained 
during  their  respective  training  programs.  These  types 
of  problems  make  it  difficult  to  attain  a  satisfactory 
level  of  standardization,  communication  and  smooth 
performance  as  an  integrated  crew. 


•  The  on-site  availability  of  full-flight  simulation  is  a 
tremendous  boost  to  conducting  a  high  quality  flight 
crew  training  program.  With  the  exception  of  one  or 
two  airplane  types  where  fleet  size  would  clearly  not 
justify  the  expense  of  dedicated  simulators  the 
operators  visited  often  stretched  themselves  financially 
to  have  a  full  flight  simulator  for  each  type  in  the  fleet. 
In  the  few  cases  where  the  simulators  were  not  owned 
the  operators  leased  simulators  and  provided  the 
instructors  from  their  own  ranks.  For  the  12  operators 
the  simulator  types  owned  as  a  percentage  of  fleet 
airplane  types  was  90%.  As  a  matter  of  comparison 
the  operators  who  have  worse  than  average  accident 
rates  have  only  a  60%  coverage  of  the  fleet  with 
company-owned  simulators. 


Recurrent  Training 


The  criticality  of  recurrent  training  to  continued  safe 
operation  is  universally  recognized.  Many  of  the  issues 
discussed  in  other  sections  of  this  article  are  implemented 
through  a  training  program  that  recognizes  and  provides 
emphasis  on  accident  related  operational  topics  such  as 
windshear,  ground  proximity  warning  system  (GPWS), 
rejected  takeoff  and  human  factors. 


All  operators  stressed  the  importance  of  a  well  managed 
and  thorough  recurrent  training  program.  Most  of  the 
operators  surveyed  train  captains  and  first  officers  to  the 
same  standards  and  provide  the  same  quality  and  quantity 
of  training  to  both.  Half  of  the  operators  also  type-rate 
all  pilots  to  the  same  standards.  A  few  provide  simulator 
training  four  times  per  year.  One  operator  who  had 
established  a  standard  of  four  recurrent  training  sessions 
per-year  per-crew  attempted  to  cut  training  costs  by 


reducing  the  training  sessions  to  two  per-year  for  crew¬ 
members  who  were  located  at  a  remote  domicile.  After 
a  trial  program  this  operator  reverted  to  the  basic  four 
sessions  per  year  based  on  a  noticable  degradation  of 
piloting  skills  in  the  two-per-year  crews.  This  skill  loss  was 
particularly  noticeable  for  the  lower  20  percent  of  the 
pilots. 

Firm  rules  exist  and  are  enforced  if  poor  performance  is 
noted  in  training.  No  flying  is  permitted  until  satisfactory 
performance  is  demonstrated.  If  problems  persist,  high 
level  management  faces  up  to  the  tough  decisions  of 
alternate  assignments,  demotion  or  dismissal. 

Several  operators  regularly  cycle  training  captains 
through  a  line-flying  period  to  keep  them  in  touch  with 
actual  operations. 

Flight  Path  Control 

Since  the  approach  and  landing  phase  accidents  comprise 
such  a  large  segment  of  the  total  accident  picture 
(Figure  3)  it  is  not  suprising  that  we  found  much  emphasis 
being  placed  on  the  stabilized  approach  and  proper  energy 
management.  It  is  felt  that  reduced  flap  settings  and  late 
gear  extension  brought  about  by  fuel  conservation 
measures  contribute  to  destabilized  approaches.  The  more 
modern,  aerodynamically  clean,  airplanes  can  pose 
problems  for  a  crew  that  does  not  plan  the  approach 
sufficiently  in  advance.  Speed  and  altitude  can  be  difficult 
to  bleed  off  if  the  ideal  approach  path  is  overshot  and/or 
airspeed  is  too  high.  Higher  thrust  settings  for  engine 
anti-ice,  if  required,  make  matters  that  much  worse.  ATC 
often  can  place  the  airplane  close-in  and  high  and,  if  the 
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Figure  3.  The  flight  segment  distribution  of  all 
commercial  accidents  between  1959-1984. 


crew  is  not  alert,  the  speed  which  must  be  dissipated  may 
preclude  attaining  a  stabilized  landing  configuration  at  a 
reasonable  altitude.  If  the  approach  is  not  stabilized  it  is 
more  difficult  to  detect  the  effects  of  windshear  and  in  fact 
any  changing  wind  conditions.  When  the  approach 
becomes  hurried  normal  monitoring  and  crew 
coordination  are  subject  to  breakdown  and  the  stage  is  set 
for  a  landing  accident.  Management  and  training  emphasis 
was  found  in  the  following  areas: 

•  There  are  firm  rules  for  flap  and  gear  extension  versus 
altitude.  There  is  often  a  cutoff  altitude  specified 
where,  if  the  airplane  is  not  on  glide  path  in  landing 
configuration  with  speed  stabilized,  a  go-around  is 
mandatory.  These  altitude  gates  are  usually  different 
for  instrument  and  visual  conditions. 

As  an  example,  one  operator  who  places  stringent 
management  controls  on  approach  techniques  begins 
with  the  simulator  phase  of  transition  training. 
Parameters  for  control  of  the  approach  must  be 
committed  to  memory.  For  given  landing  configura¬ 
tions  the  approximate  attitude,  thrust  setting  and 
vertical  speed  targets  are  committed  to  memory  and 
applied  so  that  the  following  rules  for  the  approach 
can  be  met: 

1.  The  airplane  should  be  stabilized  in  landing 
configuration  by  1000  feet  above  touchdown.  If 
there  is  significant  crosswind,  low  clouds  or  poor 
visibility  then  the  airplane  should  be  stabilized  by 
at  least  the  outer  marker. 

2.  It  must  be  stabilized  by  500  feet  and  if  not,  an 
immediate  missed  approach  must  be  initiated. 

•  There  is  a  requirement  for  monitoring  instruments 
including  approach  displays  during  visual  approaches. 
One  operator  has  a  policy  that  whenever  landing  aids 
are  available  every  approach  is  an  instrument 
approach.  Operational  procedures  and  training 
reinforce  the  philosophy  that  the  ability  to  see  the 
ground  is  just  an  additional  landing  aid. 

•  The  infrequent  occurrence  of  the  visual  missed 
approach  seemed  statistically  unreasonable  to  some 
operators  and  prompted  them  to  emphasize  the 
go-around  decision  in  training,  remove  the 
requirement  to  report  go  arounds  to  management,  and 
develop  procedures  which  encourage  the  crew  to  make 
a  go-around  rather  than  attempt  a  landing  out  of  a  poor 
approach. 

•  One  operator  felt  that  early  training  (primarily 
military)  over-emphasized  the  skills  for  a  complete 
power-off  approach  and  diminished  attention  to 
proper  approach  techniques. 
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First  Officer  Flying  Rules 


Some  operators  believe  that  captains  do  not  always  do  the 
monitor  functions  well  when  the  first  officer  is  flying.  This 
problem  receives  special  training  emphasis,  to 
demonstrate  that  the  captain  in  this  circumstance  really 
must  perform  all  the  first  officer  duties  in  addition  to  his 
pilot-in-command  functions.  (To  better  understand  the 
individual  crew  member’s  involvement  in  this  problem, 
Boeing  studied  crew  caused  accident  history  where  it 
could  be  determined  which  pilot  was  handling  the 
airplane.  It  was  found  that  in  a  majority  of  the  accidents 
the  non  flying  pilot  (NFP)  had  the  opportunity  to  effect 
corrective  action  but  failed  to  do  so.  However,  the  captain 
did  not  appear  to  be  an  ineffective  NFP  when  compared 
to  the  first  officer.) 

There  are  usually  specific  rules  governing  when  the 


opportunity  for  error  in  a  realistic  flight  that  has  an 
abundance  of  high  crew  workload  and  stress.  Some 
operators  see  a  need  for  more  realistic  scenarios  with  less 
emphasis  on  multiple  failures  in  non-critical  phases  of 
flight  and  more  emphasis  on  realistic  situations  in  the 
approach  and  landing  phases.  Creating  enough  scenarios 
to  adequately  cover  the  training  objectives  and  to  prevent 
crew  familiarity  through  repetition  was  a  frequently 
mentioned  problem. 

Reference  Data 

Some  operators  are  concerned  that  automated  dispatch, 
computerized  airport  analysis  and  flight  planning 
although  considered  essential  in  today’s  operation  may 
reduce  the  pilots'  familiarity  with  the  performance 
capability  of  the  airplane.  One  operator  is  increasing 
emphasis  on  this  aspect  in  the  training  program. 
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captain  must  make  the  approach  and  landing.  These  vary 
widely  from  operator  to  operator  but  many  had  rules  more 
specific  than  Captain 's  choice  or  alternate  legs.  These  rules 
which  specifically  call  for  the  captain  to  handle  the 
airplane  fall  into  four  general  categories: 

•  Airports  that  have  geographic  hazards  or  congestion 
problems. 

•  Weather  minimums  -  most  frequently  below  1200 
RVR  or  for  Category  II  and  III  landings. 

•  When  there  is  a  primary  airplane  system  malfunction. 

•  Low  first  officer  time  in  type. 

One  operator  determined  that  ground  handling  accidents 
and  incidents  were  more  frequent  when  the  first  officer 
was  taxiing.  A  rule  was  adopted  that  requires  the  captain 
to  taxi. 

Line  Oriented  Flight  Training  (Loft) 

LOFT  was  fully  implemented  in  the  recurrent  training 
program  for  nine  of  the  operators  and  partially 
implemented  at  two.  Those  operators  who  use  LOFT  were 
of  the  opinion  that  this  training  element  is  a  very  important 
safety  enhancement.  It  provides  a  method  of  exposing  a 
crew  to  a  complete  flight  operation  (such  as  preflight 
operations,  the  flight  itself  and  the  postflight  activity)  in 
such  a  way  that  company  procedures,  flight  procedures, 
flying  techniques  and  cockpit  management  (human 
factors)  are  observed  without  providing  any  more 
assistance  to  the  crew  than  it  would  have  on  a  real  flight. 
Crew  operating  and  teamwork  shortcomings  are  reviewed 
in  a  debriefing  by  a  training  pilot.  Seme  airlines  have 
introduced  video  recording  playback  into  the  debriefing 
and  found  this  beneficial  in  allowing  self-critique  by  the 
pilots. 

Thoughtful  preparation  of  LOFT  scenarios  is  essential  in 
order  to  provide  a  maximum  of  crew  decisions,  actions  and 


More  on  Management 

The  key  safety  elements  discussed  so  far  were  recognized 
by  management  as  significant  safety  concerns  upon  which 
they  placed  almost  universal  emphasis  in  each  of  their 
respective  flight  training  programs.  The  items  below  are 
more  random  in  nature  and  do  not  lend  themselves  to 
catagorization  but  they  too  fit  the  mold  of  key  elements. 

•  Flight  data  recorder  monitoring  is  being  practiced  by 
3  of  the  12  operators  primarily  as  a  tool  to  provide 
direction  to  the  training  department.  The  digital 
recorder  tapes  for  each  flight  are  compared  by 
computer  against  predetermined  standards  for  specific 
flight  phases.  Substantial  deviation  from  standards 
such  as  early  rotation,  high  descent  rate,  unstabilized 
approach  and  high  touchdown  speeds  are  automati¬ 
cally  identified  for  further  investigation.  The  operators 
who  have  implemented  this  system  are  very  enthu¬ 
siastic  about  its  contribution  to  safety  and  have  found 
ways  to  structure  the  program  so  that  flight  crew 
concerns  are  addressed  and  their  support  is  obtained. 
Taking  a  serious  flight-related  event  to  the  specific 
flight  crew  for  review  is  done  through  the  Association 
representative  in  the  few  instances  it  is  required,  but 
this  aspect  is  considered  of  far  less  significance  in 
comparison  to  the  benefits  derived  in  understanding 
where  operational  procedures  and  or  the  training 
program  might  need  modification  or  emphasis. 

•  All  operators  have  firm  policies  requiring  immediate 
pull-up  in  response  to  a  ground  proximity  warning  in 
Instrument  Meteorological  Conditions  (IMC)  but  only 
two  investigate  each  case  in  order  to  determine  the 
cause  and  verify  that  a  proper  response  was  made. 
Most  operators  express  concern  that  unreliability  of 
the  early  GPWS  is  still  influencing  some  pilots  to  delay 
pull-up  instead  of  initiating  an  immediate  response. 
Several  operators  have  already  implemented  GPWS 
response  training  in  their  simulator  programs. 
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•  Pilots  arc  being  made  more  aware  of  the  significance 
of  Vj  and  the  practical  aspects  related  to  aborting  or 
continuing  a  take  off.  The  criticality  of  actions  to  stop 
or  to  go  when  airplane  speed  is  in  the  vicinity  of  V  | 
is  becoming  better  understood  by  the  flight  crews  in 
these  airlines. 

•  Problem  airports  receive  careful  analysis  and 
management  attention;  special  training,  experience 
and  currency  requirements  are  laid  down.  The  method 
of  handling  this  problem  varies  from  operator  to 
operator.  As  an  example  one  operator  classifies  all 
airports  of  interest  into  three  categories  and  specifies 
certain  landing  requirements  for  each  category.  These 


1.  Standard  -  all  captains  are  qualified. 

2.  Operational  problems  -  24  months  with  the 
airline  as  captain  or  prior  operation  into  the 
airport  as  pilot  or  observer. 

3.  Non-standard  -  as  pilot  or  observer  within  last 
12  months.  In  addition  a  number  of  the  non¬ 
standard  airports  have  more  stringent  require¬ 
ments.  One,  for  example  includes: 

o  First  checkout  with  a  training  pilot 

o  At  least  5  takeoffs  and  landings  before  revenue 
flight 

o  Minimum  of  2  approaches,  landings  and 
takeoffs  within  the  last  2  months 

o  Landing  by  captain  only 

o  Night  checkout  prior  to  night  operations 
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Where  do  we  go  from  here? 

The  results  of  this  limited  study  indicate  that  the  potential 
for  improvement  of  the  crew  caused  accident  rate  may  be 
great  if  some  key  policies,  procedures  and  practices  are 
adopted  and  followed.  Figure  4  shows  that  this 
improvement  might  approach  a  factor  of  ten  for  the 
worldwide  fleet  if  the  rate  for  the  12  operators  is 
statistically  valid.  A  review  of  the  nearly  350  Boeing 
operators  shows  that  at  least  54  have  accident  rates  worse 
than  average  and  therefore  might  benefit  from  a  close  look 
at  their  own  standards,  discipline,  procedures  and  flight 
crew  training  programs  (Figure  5). 

Boeing  will  continue  to  investigate  and  expand  its 
knowledge  of  crew  caused  accidents  and  communicate  any 
findings  to  the  operators. 

To  summarize,  we  are  unable  to  prioritize  the  findings  of 
this  study  since  every  aspect  is  important.  However,  the 
low  crew  caused  accident  carriers  seem  to  have  found  the 
key  to  integrating  the  elements  into  a  successful  operation. 

Our  sincere  thanks  go  to  the  38  management  pilots  of  the 
12  airlines  that  gave  so  generously  of  their  time  and  talents 
to  make  this  first  step  possible.  Comments  from  readers 
of  this  article,  particularly  more  Key  Elements  for  safe 
operation,  would  be  most  welcome.  Please  write  to: 

Boeing  Commercial  Airplane  Co. 

Attn:  Safety  Manager  M/S  98-33 
P.O.  Box  3707 
Seattle,  WA  98124 

Flight  operations  and  training  managers  who  are  willing 
to  provide  details  of  their  airline’s  operation  in  order  to 
broaden  the  scope  of  this  study  may  request  a  copy  of  a 
questionnaire  developed  for  this  purpose. 
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EXTRACT  FROM  DOUGLAS  AIRCRAFT  COMPANY  LETTER 


During  the  second  Workload  Workshop,  held  in  Seattle  during 
September,  various  data  collected  during  our  PART-Task  simulation 
was  presented.  Data  was  presented  for  the  Subjective  Workload 
Assessment  Technique  (SWAT)  and  the  NASA  Task  Load  Index  (TLX) 
which  contained  some  errors.  The  following  proceedings  contains 
the  correct  data,  and  data  analyses  for  both  validity  and 
reliability.  None  of  the  d i s c r i mi na b i 1 i ty  analyses  outcomes 
changed  for  the  correct  data.  The  ability  to  discriminate  LOW 
and  HIGH  workload  remained  the  same,  as  well  as  the 
discrimination  among  individual  "windows"  of  measurement  and 
discrimination  among  phases  of  flight  within  LOW  and  HIGH 
workload,  re s pe c t i ve 1  y .  The  reliability  test/retest  data  was 
affected  significantly  by  the  analyses  done  with  the  correct 
data.  SWAT,  which  did  not  appear  to  be  very  reliable  from  one 
session  to  the  next,  turned  out  to  be  nearly  as  reliable  as  the 
NASA-TLX  data.  TLX  reliability  coefficients  slipped  very 
slightly  with  the  analyses  redone  with  the  correct  data.  The  two 
measures,  SWAT  and  TLX,  appear  to  be  more  similar  than  originally 
presented  at  the  Seattle  workshop. 
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INTRODUCTION 


Currently,  The  Boeing  Company  and  Douglas  Aircraft  Company  are  jointly 
working  a  USAF/FAA  sponsored  workload  contract.  The  objective  of  this 
contract  is  to  provide  guidelines  for  a  flexible  basis  for  selecting  workload 
measures  in  future  years  as  flight  deck  requirements  change.  This  will  enable  the 
FAA  to  evaluate  workload  measurement  plans  for  crew  size  substantiation  and 
workload  acceptability  during  aircraft  certification  efforts.  Guidelines  will  be 
provided  for  selecting  workload  measures  that  are  valid,  reliable,  and  applicable 
for  airplane  certification. 

*  The  manufacturers  of  commercial  transport  aircraft  are  aware  of  the  special 
responsibilities  and  caution  needed  to  assure  a  safe  aircraft  design.  The  technical 
approach  for  this  contract  has  emphasized  established  scientific  methodology 
and  practical  application  of  methods  in  the  selection  of  workload  measures  for 
aircraft  certification.  A  step-by-step  procedure  has  been  used  for  identifying 
candidate  workload  measures  and  then  confirming  the  validity,  reliability,  and 
applicability  of  the  candidates  by  means  of  rigorous  aircraft  simulation  testing. 

Tne  procedures  used  have  enabled  quantitative  descriptions  to  be  made  of  the 
statistical  validity  and  reliability  of  the  workload  measures  employed  in  testing  to 
date.  —  ^  {4  ')  4, 

Results  from  the  part-task  simulation  testing  phase  of  the  contract  were  presented 
at  a  two-day  workshop  held  in  Seattle,  Washington,  on  September  15  and  16  at 
the  Crowne  Plaza  Hotel.  Eighty  attendees  were  drawn  from  a  wide  cross  section 
of  operational  personnel  ana  potential  workload  measurement  users.  University 
scientists  from  the  first  workshop  held  In  Long  Beach,  California,  on  February  24 
and  25, 1987  were  invited  to  attend  to  help  assess  the  scientific  quality  of  the 
results  and  the  appropriateness  of  the  conclusions  drawn  from  the  data  and  test 
designs.  The  remainder  of  the  attendees  were  from  aerospace  industry, 
government  regulatory  agencies,  military  workload  experts,  and  NASA. 

Attendees  were  asked  to  provide  comments/discussion  regarding  their  views  of 
the  part-task  testing  results.  They  were  also  asked  to  comment  on  the  validity  of 
the  data  and  what  the  data  suggested  regarding  the  applicability  of  the  candidate 
workload  measures  to  aircraft  certification.  Audience  comments  were  reviewed 
and  incorporated  prior  to  the  full-mission  testing  phase  of  the  contract.  Included 
in  these  proceedings  are  copies  of  all  the  workshop  presentations,  results  of 
analysis  of  the  data  collected  in  part-task  testing  and  the  results  of  the  time-line 
analysis.  In  addition,  a  summary  of  the  discussions  held  over  the  two  days  is 
included. 
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PROGRAM  -  WORKSHOP  #2 

September  15  and  16, 1987 

DAY  1 


T!M£ 

CARLSBAD  ROOM 
08:30-09:00  am 

09:00-09:10  am 

09:10-09:30  am 

09:30-09:40  am 
09:40-10:10  am 
10:10-10:40  am 

10:40-11:00  am 
1 1 :00-1 1 :20  am 

11:20-1 1:50  am 

11:50-01:00  pm 


IYENT 


Registration  (Continental  breakfast  available) 
(Nancy  Whitman) 

Welcome  from  USAF  sponsor 
(Harry  Britten-Austin) 

Welcome/opening  remarks  from  Boeing 
(Brien  Wygle-  Vice  President  Flight 
Operations) 


Workshop  objective 
(George  Boucek) 

Opening  remarks  from  FAA  sponsor 
(Pete  Hwoschinsky) 

Program  overview:  Validity,  reliability  criteria  for  the 
selection  of  measures 
(Mike  Biferno) 

Break 

Application  of  data 
(George  Boucek) 

Description  of  part-task  simulation 
(Diane  Sandry-Garza) 

Luncheon 
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TIME 

CARLSBAD  ROOM 
01:00-01 :40  pm 


01:40-02:20  pm 


02:20-02:40  pm 
02:40-03:20  pm 


03:20-04:00  pm 


04:00-05:00  pm 


05:00  pm 
06:00  pm 


10:00  pm 


PROGRAM  -  WORKSHOP  #2 

September  15  and  16, 1987 

DAY  1  (Continued) 


EVENT 


Subjective  workload  measurement:  A 
review  of  the  data  regarding  validity, 
reliability,  applicability 
(Bill  Corwin) 

Physiological  workload  measurement:  A 
review  of the  data  regarding  validity, 


reliability,  applicability 
(Sam  Metalis) 


Break 

Performance  workload  measurement:  A 
review  of  the  data  regarding  validity, 
reliability,  applicability 
(Bill  Corwin) 

Analytical  workload  assessment:  A 
description  of  the  TLA  task  and  a 
review  of  the  data 
(Aileen  Logan) 

Question/Answer  Session 
(George  Boucek,  Mike  Biferno) 

Adjourn 

Regroup  in  lobby  for  bus  to  Fisherman’s 
Terminal  (evening  includes  a  cruise 
and  dinner  buffet) 

Return  to  Crowne  Plaza 
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PROGRAM  -  WORKSHOP  #2 

September  15  and  16, 1987 

1 

DAY  2 

I 

TIME 

EVENT 

1 

CARLSBAD  ROOM 

08:30-09:00  am 

Continental  breakfast  available 

1 

09:00-10:30  am 

Comparisons  of  workload  measures 
(Panel) 

a 

10:30-10:50  am 

Break 

| 

10:50-1 1:20  am 

Description  of  full  mission  simulation 
(Diane  Sandry-Garza,  Bill  Corwin) 

i 

11:20-12:20  am 

Discussion  of  full  mission  simulation 
(George  Boucek,  Mike  Biferno) 

i 

12:20-12:30  pm 

Concluding  remarks 
(George  Boucek,  Mike  Biferno) 

u 

12:30  pm 

Luncheon  Buffet 

WELCOME  AND  OPENING  REMARKS  FROM  THE  USAF 
Sqn.  Ldr.  HARRY  BRITTEN-AUSTIN 
Wright-Patterson  Air  Force  Base 


Edited  Transcript 

Good  morning  Ladies  and  Gentlemen,  welcome  to  the  second  Workshop 
of  this  Crew  Workload  Measurement  program.  It’s  nice  to  see  so  many  of  you 
made  it  here.  We  obviously  gave  you  a  good  time  at  the  first  workshop.  Many  of 
you  worked  very  hard  at  that  first  workshop,  and  I  hope  you’re  going  to  be 
pleased  with  the  results  you’ll  see  today  and  tomorrow  which  are  a  direct 
consequence  of  your  efforts  in  Long  Beach  in  February.  In  addition  to  those  who 
attended  in  February,  we  have  many  user  groups  represented  here  today 
because  we  feel  the  flavor  of  the  program  has  now  drifted  toward  implementation 
of  the  measurement  techniques.  I’m  pleased  to  see  so  many  representatives 
from  the  FAA,  APA,  ALPA,  AtA,  from  NASA,  the  academic  community,  and  even 
from  overseas. 

I’ve  just  been  asked  to  introduce  the  key  players  in  the  program.  First,  my  co¬ 
sponsor  from  the  FAA,  Pete  Hwoschinsky,  and  the  two  people  who  manage  the 
contract,  Mike  Biferno  from  Douglas  Aircraft  and  George  Boucek  from  The  Boeing 
Company. 

I  think  you’ll  find  it  an  interesting  couple  of  days.  We  have  some  special  diversion 
planned  for  you  as  well.  This  evening’s  cruise  promises  to  be  very  entertaining. 
We  were  well  looked  after  at  the  social  last  night,  and  we’re  hoping  you’ll  have  two 
very  pleasant  days  here.  Thank  you. 


WELCOME  AND  OPENING  REMARKS  FROM  THE  BOEING  COMPANY 

BRIEN  WYGLE 


Vice-President  Flight  Operations 
Boeing  Commercial  Airplane  Company 


Edited  Transcript 

I’d  like  to  welcome  everyone  to  Seattle  from  The  Boeing  Company.  I  went 
to  the  cocktail  party  last  night  and  enjoyed  it  very  much.  I  hope  you  all  have  a 

§ood  stay.  Since  we've  had  a  very  dry  summer,  and  we  knew  you’d  like  it  wet  in 
eattle  like  it’s  supposed  to  be,  we’ve  even  provided  rain. 

I’m  flattered  to  be  asked  to  address  this  erudite  gathering  on  the  subject  of 
workload.  We  have  been  dealing  with  the  subject  now  for  a  very  long  time.  We 
owe  thanks  to  the  SAE  S-7  committee,  for  example,  and  many  other  organizations 
and  engineering  designers  for  providing  us  with  todays  cockpits.  Certainly  we 
have  come  a  long  way. 

I  think  the  reason  we  have  such  good  attendance  and  why  this  is  an  important 
session  is,  in  fact,  because  today  you  are  where  the  action  is.  What  you’re  doing 
now  is  what’s  important.  We  have  learned  a  lot  over  the  years,  but  the  study  of 
workload  and  human  factors  and  their  role  in  preventing  accidents  is  certainly  a 
wave  of  the  future.  That  is  what  we  must  devote  ourselves  to,  perhaps  in  a  more 
informed  and  less  hysterical  manner,  than  in  the  past. 

In  presenting  accident  statistics  there  is  always  a  classic  slide  which  shows  that 
accidents  occur  most  often  in  approach  J  landing.  I  want  to  point  out, 
however,  that  we  mustn’t  make  too  rr  3ny  eariy  assumptions.  I  will  refer  to  two 
fairly  recent  accidents. 

First,  the  747  acciden*  in  Japan,  one  of  the  most  tragic  and  terrible  accidents  of  all 
times,  was  not  due  to  the  crew  s  failure  There  was  little  the  crew  could  do  to  save 
themselves.  Certainly  one  of  the  most  terrifying  things  to  a  pilot  outside  of  in-flight 
fire,  is  to  be  left  with  a  hydraulically  controlled  airplane  with  no  hydraulics  to 
control  surfaces.  That  accident  nonetheless  was  due  to  human  failure.  The 
original  problem  was  an  improper  repair  done  to  the  rear  bulkhead  which  resulted 
in  material  failure.  It  is  a  long  and  complex  story  of  a  series  of  minor  human  errors 
which  led  to  fatigue  failure  and  a  catastrophic  accident.  To  trace  and  correct 
those  things  is  very  complicated  because  it’s  not  just  human  failure  and  its  not 
just  the  engineering  of  the  job.  We  have  to  see  why  people  fail,  why  this  accident 
was  not  prevented. 

The  second  is  the  more  recent  MD-80  accident  in  which  it  appears,  although  I 
don’t  like  to  second  guess  the  NTSB,  that  the  flaps  were  not  extended.  It  has 
been  suggested  that  there  was  no  warning  horn,  and  I  suppose  that  we’ve 
learned  to  depend  on  those  warning  horns  to  tell  us  that  we're  configured  for  take 
off.  There  are  only  a  few  things  that  you  have  to  be  certain  about  during  take  off. 


One  is  your  flaps.  Nonetheless,  if  the  horn  had  been  pulled  by  the  circuit  breaker, 
because  we’ve  all  seen  warning  horns  silenced,  we  have  to  ask  ourselves  why  the 
crew  would  do  that.  We  must  not  just  talk  to  the  crew  and  tell  them  not  to  pull  the 
circuit  breaker.  We  have  to  look  at  “why"  they  pulled  it.  Where  in  our  design  did 
we  permit  that  to  occur?  That’s  the  thing  that  the  designers  are  asking  themselves 
every  day. 

I  would  like  to  tell  you  a  little  about  a  study  done  at  The  Boeing  Company  recently 
by  two  of  our  engineers,  Les  Lautman  and  Peter  Gallimore  (included  in  this 
mailing  is  a  complete  copy  of  this  study).  They  noted  that  in  line  with  Pareto’s 
thinking  and  Pareto’s  charts,  15%  of  the  world’s  airlines  had  80%  of  the  accidents 
and  80%  of  the  airlines  had  no  accidents.  They  took  a  recent  10  year  period  and 
looked  at  Boeing  customers  only.  They  thought  we  should  be  working  on  the 
concentration  of  where  the  accidents  are.  That’s  a  little  difficult  to  do  however.  It’s 
not  the  manufacturer’s  job  to  tell  the  airlines  how  to  prevent  accidents,  and 
manufacturer’s  are  not  too  well  received  when  they  attempt  to  do  so.  Lautman 
and  Gallimore  decided  to  visit  the  airlines  that  traditionally  don’t  have  accidents  to 
see  what  those  airlines  did  in  flight  operations.  I’m  sure  there’s  room  for 
significant  statistical  error,  but  they  came  up  with  a  representative  number  of 
airlines,  both  domestic  and  international,  that  had  good  safety  records.  They 
asked  those  airlines  what  they  did  in  flight  operations,  and  listed  all  these  things  to 
see  what  these  airlines  did  in  common.  Not  surprisingly,  the  airlines  had  a  great 
deal  in  common. 

First,  they  had  a  well  organized  and  disciplined  flight  operations  department.  They 
had  people  who  were  paying  full-time  attention  to  the  administration  of  the 
department.  They  had  a  corporate  attitude  of  flight  safety,  because  we  know  if 
your  trying  to  run  a  flight  operations  department  and  you  can’t  get  the  budget  you 
need  from  the  president  and  the  chief  financial  officer ,  you  cannot  provide  the 
needs  for  a  safe  flight  operations.  In  all  cases  the  corporate  management  was 
safety  conscious.  There  was  sufficient  response  from  the  bean  counters  and 
other  non-flying  people  that  they  recognized  the  needs  and  provided  the  means 
to  have  a  safe  flight  operation. 

Another  factor  in  common  was  the  concept  of  standardization  pilots.  That  is,  there 
was  a  pilot  assigned  to  be  the  standardization  pilot.  He  would  see  that  all  the  flight 
crews  were  using  the  same  procedures.  They  had  a  check-airman  program 
where  the  line  pilot  was  continually  checked,  in  many  cases  beyond  the 
governmental  requirements.  They  had  mandatory  briefings  before  takeoff  and 
before  approach.  This  was  a  procedure  pilots  had  to  go  through  no  matter  how 
routine  it  seemed.  The  pilot  was  required  to  brief  before  take  off  and  before 
approach,  including  the  missed  approach.  It  is  this  discipline,  we  believe,  that 
contributes  to  their  safety. 

Another  common  factor  among  these  airlines  was  that  they  all  owned  their  own 
simulators.  They  did  very  little  contract  training.  If  they  had  a  small  fleet  of  a 
certain  type  of  aircraft  that  didn’t  justify  a  simulator,  they  might  contract  with 
someone  else,  but  they  would  send  their  own  instructors  and  would  use  their  own 
procedures.  They  developed  standardized  approaches  with  the  same 
procedures  for  callouts,  the  same  requirements  for  a  stabilized  approach  -  on 
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speed,  on  glide  slope  and  so  on.  Each  of  these  companies  tightly  controlled  the 
cockpit  procedures  and  the  cockpit  language.  There  was  no  room  for  casual 
language  usage.  The  crew  had  to  call  everything  by  its  right  name.  The  training 
was  extensive,  in  many  cases  more  than  the  requirements.  Most  often  these 
airlines  used  the  LOFT  concept  of  line  oriented  flight  training. 

It  has  been  my  experience  that  most  of  the  things  I’ve  mentioned  tend  to  be 
missing  or  improperly  applied  in  those  airlines  that  have  a  bad  safety  record.  I'm 
sure  my  friend  Mr.  Knickerbocker  from  Douglas  Aircraft  would  agree  from  his 
observation  of  customer  operations.  We  know,  in  fact,  how  to  create  a  safe 
airline.  It  is  not  always  done  however,  for  many  reasons. 

I  believe  the  modern  cockpit  to  be  a  very  good  one.  I  think  starting  with  the 
757/767  and  A310,  to  be  followed  by  the  MD-80  advanced  and  737  advanced 
cockpits  and  followed  by  the  A320,  MD1 1  and  747-400,  these  cockpits  are  really 
very  much  the  same  idea.  There  are  detail  differences  but  everyone  has  a  similar 
concept  of  what  a  modern  cockpit  is.  I  think  we  would  all  agree  that  the  three 
man  cockpit  is  an  out  of  date  and  unsatisfactory  cockpit  in  the  world  today.  That 
was  certainly  controversial,  but  once  put  to  bed  that  idea  has  swept  widely  across 
the  industry.  In  fact  two  men  make  an  ideal  human  relation  combination  to 
operate  an  airplane. 

There  is  criticism  of  the  cockpits  today,  that  we  have  provided  an  environment 
where  the  pilot  may  become  overly  complacent  and  not  be  sufficiently  motivated 
to  do  the  job  properly,  resulting  in  errors.  There  may  very  well  be  justification  for 
this.  However,  there  are  other  times  when  the  pilot  sees  no  evidence  of  insufficient 
workload.  For  instance,  when  the  pilot  gets  an  approach  change  at  the  last 
minute  and  is  making  inputs  into  his  flight  management  computer,  I  think  those 
are  times  when  he  is  quite  heavily  loaded.  Certainly,  if  we  can  simplify  the  number 
of  keystrokes  required  to  input  an  approach  change,  that  would  be  a  valuable 
improvement.  Of  course,  people  are  working  on  that.  We  certainly  have  to 
address  the  subject  of  workload;  where  it  is  high  and  where  it  isn’t  sufficiently 
high.  The  idea  of  keeping  people  alert  is  an  important  one. 

The  concept  of  fatigue  is  also  important.  None  of  us  work  very  well  when  we’re 
on  the  verge  of  falling  asleep.  The  fatigue  problem  is  a  very  difficult  one  to 
approach.  No  matter  how  complicated  we  make  our  duty  time  allowables,  we’re 
never  going  to  really  know  the  state  of  that  pilot  when  he  gets  into  the  airplane. 

We  assume  if  a  pilot  leaves  at  9  o'clock  in  the  morning  and  is  bright  and  rested 
and  in  good  shape,  he  can  fly  a  long  time.  I  suppose  that’s  generally  true,  but  we 
know  that  sometimes  people  have  a  bad  night,  or  sometimes  they  have  mental 
stress.  We  also  know  that  some  pilots  who  take  off  at  6  o’clock  in  the  evening  are 
very  fresh  because  they’re  those  lucky  people  who  can  lay  down  and  sleep  4 
hours  in  the  afternoon.  Not  all  of  us  can  do  that.  So  we  really  don’t  know  the 
state  of  the  pilot  when  he  gets  into  the  cockpit.  We  really  don’t  know  what  an 
individual’s  mental  and  physical  state  will  permit  him  to  do  on  that  particular  day.  I 
certainly  don’t  know  how  to  measure  that,  but  perhaps  it’s  food  for  thought. 

Finally,  I  would  like  to  comment  on  the  commitment  of  The  Boeing  Company  to 
this  particular  group  and  to  the  subject  of  workload  and  human  factors  in  general. 
We  have  combined  our  pilots,  engineers  and  human  factors  people  into  a  group 


that  is  jointly  responsible  for  the  development  of  flight  decks.  We  are  committed  to 
flight  deck  improvement  from  both  a  working  environment  and  a  safety  point  of 
view.  I  think  we  have  made  that  commitment,  and  it  is  clear  to  me  that  our 


correct. 

I  am  very  impressed  to  see  both  the  number  and  the  qualifications  of  the  people 
who  have  attended  this  workshop.  I  would  like  to  complement  Joe  Tymczyscyn, 
Diane  Sandrv-Garza,  and  Aileen  Logan  and  the  staff  who  have  prepared  for  this 
workshop,  and  I  hope  and  trust  that  you  will  have  both  a  pleasant  and  useful  stay. 
Thank  you. 
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OPENING  REMARKS  FROM  FAA  SPONSOR 
Peter  V.  Hwoschlnsky 
September  15, 1987 


Good  morning  ladies  and  gentlemen,  I  would  like  to  reaffirm  the  welcome  from 
our  contractors,  and  thank  all  of  you  for  taking  your  valuable  time  to  attend  this 
workshop  and  assist  us  in  providing  support  to  you  in  minimum  flightcrew 
determination. 

Let  me  take  a  few  moments  to  inform  you  how  this  project  is  organized,  who  is 
involved,  and  how  what  we  do  will  be  used.  First  or  all,  regional  requirements  are 
recognized  by  the  office  of  Aviation  Standards  in  Washington.  They  are  prioritized 
and  then  given  to  the  appropriate  R&D  office  for  action. 

The  Cockpit  Technology  Program  Office  (for  whom  I  work)  was  asked  to  develop 
this  project  in  response  to  Aviation  Standards  and  to  the  Presidents  Commission 
on  Crew  Complement.  Several  (14)  of  the  recommendations  therein  referred  to 
developing  and  implementing  improved  methods  of  measuring  aircrew  workload. 
Simultaneously,  the  FAA  Northwest  Mountain  Region  developed  a  draft  advisory 
circular  on  minimum  flight  crew  requirements  (AC  25.1523),  which  has  been 
circulated  for  comment.  In  order  to  provide  guidance  materials  for  evaluating 
compliance  with  this  circular,  we  developed  a  project  to  document  existing 
methods  of  workload  measurement,  to  organize  and  catalogue  promising  new 
techniques,  and  to  test  a  representative  sample  of  each  type  of  physiological, 
subjective,  and  performance  related  measures.  We  did  this  through  a  seed 
money  contract  from  our  Interagency  Agreement  with  the  Air  Force  to  take 
advantage  of  existing  expertise  and  manufacturers'  experience.  The  Douglas  and 
Boeing  Companies  have  agreed  to  share  part  of  the  burden  of  this  effort  through 
in  house  support. 

At  the  first  workload  workshop  which  we  held  last  February,  we  asked  an 
assemblage  of  experts  (many  of  whom  are  here  today)  to  review  a  rather  lengthy 
list  of  workload  measures  and  recommend  to  us  which  ones  we  should  consider 
for  testing  and  inclusion  in  the  Advisory  Circular.  At  the  conclusion  of  the  part 
task  and  rull  mission  simulations  and  our  evaluation  thereof,  we  will  provide  a 
report  to  the  Northwest  Mountain  Region  Aircraft  Certification  Directorate  and 
Aircraft  Evaluation  Groups;  and  FAA  Headquarters  Flight  Standards  with  our 
recommendations. 

Most  of  you  here  today  already  have  a  good  working  knowledge  of  what 
workload  measurement  is,  ana  how  it  is  used  in  design.  However,  on  the 
certification  side,  we  have  engineers  and  certification  specialists  who  are  expert 
pilots  who  use  that  expertise  to  evaluate,  by  comparison,  the  proposed  new 
systems  and  cockpit  suites  during  all  phases  of  design,  development  and 
production.  Our  intention  in  the  project  is  to  provide  additional  means  for  those 
specialists  to  gain  an  understanding  of  the  workload  demands  of  any  new  design. 
We  have  no  intention  of  replacing  the  existing  well  proven  and  highly  successful 
means  of  subjective  assessment.  We  see  this  effort  as  an  adjunct  to  the  existing 
system  when  the  advisory  circular  is  published  in  final  form,  the  certification 
process  will  continue  as  before  with  tne  addition  of  looking  at  manufacturers' 
documentation  of  acceptable  means  of  objective  workload  assessment  as 


I 


outlined  in  the  AC.  Once  we  have  made  our  report  to  Flight  Standards,  our 
portion  of  the  project  will  be  complete. 

However,  we  feel  strongly  that  our  findings  should  be  validated  through  flight  test 
verification.  Some  follow  on  R&D  should  be  performed  to  evaluate  advanced 
techniques,  and  to  provide  a  method  or  means  of  including  new  techniques  into 
the  AC  as  appropriate. 

Finally,  the  results  which  we  will  have  gathered  for  FAR  part  25  aircraft  should  be 
adapted  for  use  with  FAR  part  23  certification  use,  since  it  is  generally  conceded 
that  single  pilot  IFR  flight  in  a  small  plane  can  be  one  of  the  most  demanding 
taskloads. 

I  will  let  Douglas  and  Boeing  contractors  fill  you  in  on  the  details  of  our  project. 
George  Boucek  has  given  you  our  objectives.  Now  Mike  Biferno  will  give  you  a 
program  overview. 

Thank  you. 
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Part-Task  Simulation  Protocol 
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SUBJECTIVE  MEASURES 


All  simulation  sessions  were  video  taped.  Subjects  viewed  the  video  tapes  of  their 
flights  at  the  end  of  a  day’s  session  for  the  purpose  of  subjective  rating.  Half  of 
the  subjects  used  the  Subjective  Workload  Assessment  Technique  (SWAT)  and 
the  other  half  used  the  NASA  Task  Load  Index  (TLX). 

Both  measures,  SWAT  and  TLX,  could  discriminate  the  predicted  difference  in 
workload.  Both  measures  showed  high  INTER-RATER  reliability  as  well.  The 
measures  were  highly  correlated,  r  =  .97,  indicating  that  the  measures  are  tapping 
the  same  underlying  phenomena. 

On  the  transparencies  dots  are  used  to  denote  a  statistically  significant 
relationship. 

o  On  the  chart  including  "means1'  for  the  workload  measure  the  dot 
represents  a  significant  difference  between  low  and  high  workload  for 
a  given  measurement  window. 

o  On  the  chart  including  "correlation  coefficients"  for  the  workload 
measure  the  dot  represents  a  significant  correlation  comparing  test  to 
retest  for  a  given  measurement  window 

Following  are  the  transparencies  presented  on  the  analysis  of  the  subjective 
measures  data. 
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PHYSIOLOGICAL  MEASURES 


Data  was  collected  for  horizontal  and  vertical  eye  movement  (including  blink), 
heart  rate,  and  heart  rate  variability.  Pilots  were  instrumented  with  Beckman  mini¬ 
cup  electrodes  to  record  vertical  and  horizontal  eye  movement  and  had  Electro¬ 
cardiogram  electrodes  applied  to  the  chest  to  record  heart  rate. 

Although  the  data  demonstrates  good  reliability  for  both  TEST-RETEST  and 
INTER-RATER,  eyeblinks  per  minute  cannot  discriminate  low  and  high  workload 
conditions. 

Inter-beat  intervals  (IBI's),  which  tell  how  much  time  occurs  between  heart  beats, 
rather  than  heart  rate,  which  refers  to  how  many  beats  occur  in  a  minute,  was 
used  in  the  data  analysis.  The  MEAN  and  STANDARD  DEVIATION  of  the  IBI 
values  for  each  window  were  computed.  The  MEAN  IBI  measure  was  sensitive  to 
the  differences  between  low  and  high  workload  conditions.  In  addition  it  could 
discriminate  phases  of  flight  (between  the  measurement  windows  in  a  given 
workload  condition)  better  than  any  other  workload  measure  in  the  present  study. 
The  reliability  measures,  both  TEST-RETEST  and  INTER-RATER,  were 
consistently  high  for  the  mean  IBI  measure  of  workload.  IBI  VARIABILITY  was 
not  able  to  discriminate  the  difference  between  low  and  high  workload  conditions, 
although  a  strong  trend  exists.  TEST-RETEST  reliability  for  IBI  VARIABILITY  was 
not  as  high  as  IBI  MEAN,  nor  was  INTER-RATER  reliability  as  high  as  the  IBI 
MEAN  data. 

A  heart  rate  spectral  analysis  was  also  conducted  using  software  provided  by 
Randall  Harris  and  Allen  Pope  of  NASA-Langley.  The  Blood  Pressure 
Component,  predicted  to  decrease  when  the  pilot  is  engaged  in  a  cognitive  task, 
was  not  able  to  discriminate  the  difference  between  low  and  high  workload.  It 
was  also  poor  in  reliability  and  it  had  the  worst  performance  on  TEST-RETEST 
and  INTER-RATER  reliability  measures  of  all  the  physiological  workload 
measures.  The  Respiration  Component  was  able  to  discriminate  the  low  and  high 
workload  conditions.  Similar  to  the  other  heart  spectral  analysis  measure,  tne 
Respiration  Component  did  not  demonstrate  high  reliability  in  either  the  TEST- 
RETEST  or  INTER-RATER  reliability  measures. 

On  the  transparencies  dots  are  used  to  denote  a  statistically  significant 
relationship. 

o  On  the  chart  including  "means"  for  the  workload  measure  the  dot 
represents  a  significant  difference  between  low  and  high  workload  for 
a  given  measurement  window. 

o  On  the  chart  including  "correlation  coefficients"  for  the  workload 
measure  the  dot  represents  a  significant  correlation  comparing  test  to 
retest  for  a  given  measurement  window 

Following  are  the  transparencies  presented  on  the  analysis  of  the  physiological 
data. 
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PERFORMANCE  DATA 


Data  from  the  simulator  was  collected  during  the  measurement  windows. 
Position  information  of  the  flight  controls  (wheel,  column  and  pedals)  were 
transformed  into  a  measure  of  control  activity  labelled  Control  Input.  In  addition 
altitude  over  the  outer  and  middle  markers,  flight  director  deviation,  glideslope 
and  localizer  deviation,  and  lateral  deviation  from  runway  centerline  was  collected 
in  the  Approach  and  Landing  windows.  Since  the  measurement  windows  cover 
different  lengths  of  time,  the  control  activity  is  divided  by  units  of  time  to  yield  the 
Control  Input  measure.  This  measure  indicates  the  amount  of  significant  control 
activity  per  minute.  There  was  a  strong  indication  that  Control  Input  could 
discriminate  between  the  two  levels  of  workload. 

If  the  pilot  placed  the  plane  on  autopilot,  the  data  for  control  position  and 
consequently  Control  Input,  was  not  collected.  As  a  result  there  are 
measurement  windows  in  the  low  workload  condition  which  do  not  contain  much 
data,  making  the  comparison  of  low  and  high  workload  conditions  meaningless 
from  a  statistical  point  of  view.  Even  with  the  missing  data,  however,  there  is  still  a 
strong  indication  the  Control  Input  can  discriminate  low  and  high  workload 
conditions. 

On  the  transparencies  dots  are  used  to  denote  a  statistically  significant 
relationship. 

o  On  the  chart  including  "means"  for  the  workload  measure  the  dot 
represents  a  significant  difference  between  low  and  high  workload  for 
a  given  measurement  window. 

o  On  the  chart  including  "correlation  coefficients"  for  the  workload 
measure  the  dot  represents  a  significant  correlation  comparing  test  to 
retest  for  a  given  measurement  window 

Following  are  the  transparencies  presented  on  the  analysis  of  the  performance 
data. 
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SECONDARY  TASK  MEASURES 


A  secondary  task  was  implemented  to  measure  pilot’s  spare  capacity  (spare 
capacity  is  thought  to  be  correlated  with  workload).  The  secondary  task  was  a 
response  time  task.  ATC  would  start  a  timer  when  they  issued  calls  to  aircraft 
designated  as  positive  probes.  The  positive  probe  aircraft  were  the  pilot’s  own 
aircraft  or  another  aircraft  in  the  area.  All  other  aircraft  were  designated  negative 
probes.  The  pilots  pressed  the  push  to  talk  switch  on  the  left  side  of  the  yoke 
when  they  heard  the  call  sign  of  the  aircraft  designated  as  positive  probes  and 
this  stopped  the  timer  ATC  had  activated.  Two  measures  can  be  derived  from  the 
Secondary  Task,  response  time  and  probe  accuracy. 

A  post  hoc  examination  of  the  accuracy  of  probe  delivery  indicates  that  ATC 
personnel  did  not  deliver  the  probes  in  a  consistent  fashion  across  the  different 
subjects.  Also,  responses  which  took  longer  than  10  seconds  were  disregarded 
because  the  task  was  not  executed  immediately.  Even  with  all  the  flaws  in  the  data 
collection  there  was  a  clear  trend  for  a  discrimination  between  the  low  and  high 
workload  conditions  for  Secondary  Task  Response  Time.  No  clear  trends  for 
discriminating  low  and  high  workload  are  found  for  Probe  Accuracy.  A  statistical 
analysis  of  the  reliability  measures  is  not  possible  for  either  Response  Time  or 
Probe  Accuracy  because  of  the  discrepant  sample  sizes  due  to  the  flawed  probe 
presentation. 

On  the  transparencies  dots  are  used  to  denote  a  statistically  significant 
relationship. 

o  On  the  chart  including  "means”  for  the  workload  measure  the  dot 
represents  a  significant  difference  between  low  and  high  workload  for 
a  given  measurement  window. 

o  On  the  chart  including  "correlation  coefficients”  for  the  workload 
measure  the  dot  represents  a  significant  correlation  comparing  test  to 
retest  for  a  given  measurement  window 

Following  are  the  transparencies  presented  on  the  analysis  of  the  secondary  task 
measure  data. 
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PERFORMANCE  MEASURES 
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ANALYTIC  ASSESSMENT  TECHNIQUE 


The  analytic  assessment  technique  used  was  Time  Line  Analysis  (TLA).  TLA 
computes  the  ratio  of  time  required,  that  is,  execution  time,  to  time  available 
throughout  a  mission  scenario. 

TLA  provides  the  capability  to  analyze  complete  operating  scenarios  in  which  one 
or  more  operators  perform  specified  procedures,  each  of which  is  a  sequence  of 
individual  tasks,  within  specified  time  intervals,  over  a  prolonged  period  of  time. 
Visual,  manual,  verbal,  auditory  and  cognitive  workload  estimates  are  made  for 
each  task,  procedure,  phase  and  for  the  entire  scenario.  Analysis  was  performed 
on  the  Captain's  tasks  only. 

The  Time  Line  Analysis  task  was  used  to  identify  high  and  low  task  demand 
levels.  For  instance,  all  TLA  channels  (visual,  manual,  auditory,  verbal  and 
cognitive)  showed  a  higher  workload  level  in  the  climb  phase  of  flight  when  the 
autopilot  failed  to  engage.  When  the  engine  and  hydraulic  systems  failed,  a 
higher  level  of  workload  was  again  reflected  by  TLA.  This  higher  workload  level 
then  continued  through  the  rest  of  the  high  workload  flights  because  the  pilot  was 
handflying  the  aircraft  with  the  failure  conditions  to  a  high  speed  landing.  In 
contrast,  task  demands  in  the  takeoff  segment  of  flight  for  all  scenarios  were 
identical,  so  the  workload  level  for  the  takeoff  segment  was  hypothesized  to  be 
the  same  for  both  the  high  and  low  workload  scenarios. 

As  shown  by  the  data,  TLA  accurately  reflects  the  task  demands  of  the  scenarios. 
TLA  can  be  a  very  useful  tool  to  identify  periods  of  high  and  low  task  demands. 

Following  are  the  transparencies  presented  on  the  Time  Line  Analysis 
assessment  performed. 
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•  Control  and  indicator  data 
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Visual  Data 


Timeline  Analysis 


COMPARISON  OF  WORKLOAD 
MEASURES 


DESCRIPTION  OF  CORRELATION  MATRIX  AND  FACTOR 

ANALYSIS 


A  correlation  matrix  was  created  for  all  the  workload  assessment  techniques 
presented  at  the  workshop  including:  subjective,  physiological,  performance,  and 
time-line  analysis.  The  average  score  across  subjects  for  the  14  different 
measurement  periods  (seven  measurement  windows  for  the  high  and  low 
workload  flights)  were  used  for  the  correlation  matrix.  The  time-line  analysis 
yields  a  single  score,  for  each  body  channel,  for  each  of  the  14  measurement 
windows.  The  workload  scores  were  then  used  to  compute  the  correlation 
matrix. 

The  purpose  of  the  correlation  matrix  is  to  evaluate  how  much  the  workload 
scores  ^overlap".  If  two  workload  measures  can  both  discriminate  levels  of 
workload  and  show  reliability  but  are  not  correlated  then  the  two  measures  are 
thought  to  be  evaluating  different  aspects  of  workload. 

In  addition  to  the  correlation  matrix  a  principal  components  analysis  (factor 
analysis)  was  performed  to  try  to  reveal  underlying  common  dimensions  among 
the  various  workload  measures.  The  same  14  measurement  window  were  used 
for  the  principal  components  analysis.  A  "loading"  for  the  factor  analysis  is  a 
number  between  1 .0  and  -1 .0.  The  factor  loading  can  be  thought  of  in  terms  of  an 
absolute  value  in  that  a  score  of  0.85  or  -0.85  Toads  highly  on  the  given  factor. 
The  closer  the  factor  loading  is  to  zero  the  less  that  particular  workload 
assessment  technique  is  associated  with  the  given  factor  (factor  loadings 
between  0.25  and  -0.25  are  intentionally  left  blank). 

In  addition  to  the  correlation  matrix,  the  factor  analysis  can  show  that  two  different 
valid  and  reliable  workload  measures  assess  different  aspects  of  workload.  If  the 
workload  measures  show  validity  and  reliability  but  load  on  different  factors  in  the 
factor  analysis,  it  indicates  the  measures  are  tapping  different  aspects  of 
workload. 
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COBKBL&TKM  MATRIX 


SD  ABF  ARS  EBK  WHL  STK  PDL  SHAT  TLX  STRT  STRT* 


00 

12  1.00 


73  -20 

38  .13 

BO  -.17 


.20  1 .00 

.13  -.01  1.00 


8S  -.22 


.86 

-.01 

1 .00 

.73 

.26 

.90 

1 .00 

.04 

.61 

.17 

.47 

1.00 

.69 

-.01 

.68 

.55 

-.09 

1.00 

.62 

-.12 

.59 

.46 

-.20 

.97 

1.00 

.59 

.04 

.58 

.44 

-.33 

.60 

.57 

.18 

-.25 

.46 

.25 

-.05 

.54 

.49 

ARS 

EBK 

WHL 

STK 

PDL 

SWAT 

TLX 

VISUAL 

-.55 

MANLEFT 

-.64 

MANRIGHT 

-.56 

VERBAL 

.17 

AUDITORY 

.14 

COGN'TVE 

-.02 

.02  -.39 
.03  -.41 


VIS  MANL  MANR  VERB  AUD  COG 


VISUAL  1 .00 

MAN LEFT  .74  1 .00 

MANRIGHT  .64  .28  1.00 

VERBAL  .10  -.01  .21  1.00 

AUDITORY  .55  .25  .38  .78  1.00 

COGN'TVE  .65  .42  .49  .65  .85  1.00 


•**  Critical  correlation  values  are  r(12)= .66  or  r(12)»  -.66  are  in  bold  **• 


VARIABLE  LABELS 


HRM 

HRSD 

ABP 

ARS 

EBK 

WHL 

STK 

PDL 

SWAT 

TLX 

STRT 

STRT* 


Average  Inter-beat  Interval 

Standard  Deviation  for  Average  Inter-beat  Interval 

Mulder  Spectral  Analysis  Blood  Pressure  component  from  heart  rate 

Mulder  Speotral  Analysis  Respiration  component  from  heart  rate 

Eye  blinks  per  minute 

Wheel  control  input  (aileron)  per  minute 

Stiok  control  input  (aileron)  per  minute 

Pedal  control  input  (aileron)  per  minute 

Subjective  Workload  Assessment  Technique 

NASA  Task  Load  Index 

Secondary  Task  Reaction  Time 

Probe  Acouracy,  to  positive  probes,  for  Secondary  Task 


BOEING  TIME-LINE  ANALYSIS 


VISUAL  —  Eyes 

MANRIGHT  —  Right  side  of  body 

AUDITORY  —  Listening  by  flight  crew 


MANLEFT  —  Left  side  of  body 
VERBAL  —  Spoken  communication 
COGN'TVE —  Cognitive  channel 


i  ji  ■  ip  >  ipm 


principal  composer  analysis 

SORTED  ROTATED  FACTOR  LOADINGS  (PATTERN) 


FACTOR 

FACTOR 

FACTOR 

FACTOR 

1 

2 

3 

4 

RESPIRATION 

4 

.938 

WHEEL  INPUT 

6 

.936 

STICK  INPUT 

7 

.870 

.357 

HR  VARIABILITY  2 

.826 

.346 

SWAT 

9 

.753 

.527 

SEC.  TASK  RT 

11 

.706 

-.545 

TLX 

10 

.676 

.580 

HR  AVG.  IBI 

1 

-.627 

-.626 

• 

• 

u> 

o 

BLOOD  PRESS. 

3 

-.929 

-.281 

MANUAL  LEFT 

14 

.913 

VISUAL 

13 

.887 

.367 

PEDAL  INPUT 

8 

.664 

-.284 

.656 

COGNITION 

18 

.437 

.845 

VERBAL 

16 

.837 

AUDITORY 

17 

.253 

.836 

-.274 

SEC.  TASK  % 

12 

.306 

.674 

MANUAL  RIGHT 

15 

.427 

.371 

.517 

.453 

EYEBLINK 

5 

-.318 

.867 

VP 

5.502 

4.333 

3.898 

1.851 

The  above  factor  loading  matrix  haa  been  rearranged  so  that  the  columns 
appear  in  decreasing  order  of  variance  explained  by  the  factors.  The 
rows  have  been  rearranged  so  that  for  each  successive  factor,  loadings 
greater  than  0.500  appear  first.  Loadings  less  than  0.25  have  been 
blanked. 
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Full-Mission  Simulation 
Phases  of  Flight  and  Data  Collection 
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Full-Mission  Simulation  Scenario 

(High  Workload  Leg) 
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Subjective  Measures 
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Performance  Measures 
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DISCUSSION 
SUMMARY  POINTS 


IMPLEMENTATION  OF  A  UNIDIMENSIONAL  WORKLOAD  MEASURE: 

The  feasibility  of  using  a  unidimensional  workload  scale,  in  addition  to  the  other 
subjective  measures  taken,  was  discussed.  The  possibility  of  using  a  "Modified 
Cooper-Harper"  or  the  "Bedford  Scale "  was  addressed.  The  Bedford  Scale  is  a 
measure  of  spare  capacity.  Some  felt  the  pilots  have  trouble  assessing  their 
"spare  capacity".  The  Bedford  Scale  is  used  extensively  in  the  European 
community,  and  the  FAA  showed  a  great  interest  in  having  it  assessed  during  full- 
mission  testing.  The  Bedford  Scale  would  be  administered  post-flight  with  the  use 
of  video  tapes. 

IN-FLIGHT  RATING  ISSUE: 

Should  the  use  of  in-flight  ratings  be  incorporated  in  full-mission  simulation?  A 
number  of  suggestions  were  made  to  include  an  in-flight  subjective  measure  to 
provide  opportunity  for  a  comparison  of  in-flight  versus  post-flight  subjective 
assessment.  The  discussion  included  the  question  of  which  rating  scale  could  be 
employed  in  the  particular  scenarios  already  built  for  full-mission  testing.  SWAT 
scores  can  be  taken  verbally  and  will  work  in  the  given  scenarios.  The  workload 
involved  in  these  scenarios  precludes  the  use  of  a  "clipboard"  by  the  pilot  to  give 
his  ratings.  Correct  administration  of  NASA's  TLX  requires  a  "clipboard" 
approach.  The  question  of  when  these  ratings  woula  be  taken  in-flight  was  also 
raised.  If  the  ratings  are  requested  at  the  end  of  the  measurement  windows,  then 
pilots  may  learn  where  the  measurements  are  being  taken.  This  needs  to  be 
avoided  as  the  pilot  might  change  his  performance  (ie.,  try  harder  or  be  better)  at 
specific  times  because  he  knows  he  is  being  measured  .  It  was  suggested  that 
measurements  be  taken  at  variable  intervals,  both  during  and  outside  of 
measurement  windows.  A  final  point  was  made  that  if,  in  fact,  SWAT  ratings  are 
taken  "at  the  moment"  of  the  window  closing  then  the  scores  given  may  reflect 
only  the  workload  at  that  particular  moment  and  not  the  workload  of  the  whole 
measurement  period  {i.e.,  the  measurement  window). 

SECONDARY-TASK  ADMINISTRATION: 


The  problems  that  occurred  with  application  of  the  secondary-task  measure 
during  the  part-task  simulation  were  discussed.  The  switch  used  by  the  pilot  to 
respond  to  the  positive  probes  blanked  ATC  with  the  switch  closure.  The 
feasibility  of  using  a  different  switch  was  discussed.  A  suggestion  was  made  to 
use  verbal  responses.  It  was  concluded  that  voice  relay  is  not  a  feasible 
alternative  in  the  noisy  cockpit  environment.  The  noises  often  trigger  the  switch 
relay,  and  the  variability  using  this  technique  is  extremely  high.  The  design  of  the 
secondary  task  was  discussed  in  length.  Many  felt  the  secondary  task  technique 
couldn't  be  used  in  the  flight  test  portion  of  certification.  It  was  determined, 
however,  that  the  secondary  task  may  be  valuable  even  if  it  can  only  be  used  in 
simulation.  Even  though  it  was  handicapped  during  the  part-task  testing  by 
problems  that  were  encountered,  its  data  showed  great  promise.  The  problems 
that  occurred  in  part-task  can  be  controlled  in  full-mission  testing.  Several  other 
types  of  tasks  were  suggested  for  use  as  a  secondary  task  measure;  however, 
they  lacked  the  requirements  of  the  secondary  task  to  have  both  positive  and 


negative  probes  and  to  have  the  probes  occur  often  enough  to  gather  a  good 
base  of  data.  Since  data  is  collected  on  the  Captain  only,  the  task  must  also  be 
one  that  is  normally  performed  by  the  Captain.  The  Boeing  and  Douglas  team 
members  working  this  contract  are  currently  evaluating  all  alternatives. 

USE  OF  AUTOPILOT  ISSUE: 

Autopilot  implementation  during  part-task  simulation  was  discussed.  In  part-task 
testing,  the  use  of  autopilot  was  left  to  the  pilots  discretion,  and  thus,  its  use 
became  inconsistent  across  the  subject  population.  It  was  felt  that  a  tighter 
control  on  the  use  of  autopilot  (or  preventing  its  use  altogether)  in  full  mission 
testing  should  be  required.  Various  ways  to  accomplish  this  were  discussed, 
such  as  incorporating  additional  autopilot  training  into  differences  training  or  using 
the  confederates  in  the  cockpit  to  aid  in  consistency  of  use.  It  was  suggested  that 
if  the  autopilot  could  be  an  MEL  item  for  all  leas,  there  would  be  no  chance  of 
inconsistency  among  subjects.  It  was  agreed  that  for  full-mission  testing  the 
autopilot  would  be  MEL  for  all  legs. 

INCAPACITATION  ISSUE; 

It  was  submitted  that  during  full-mission  simulation  testing  the  first  officer  might  be 
incapacitated  during  one  leg  of  the  scenario.  The  question  was  raised  whether 
the  first-officer  is  being  incapacitated  in  order  to  study  the  "incapacitation*  issue  or 
to  manipulate  the  Captain's  communications  workload.  A  discussion  on  which 
approach  is  more  relevant  for  certification  ensued.  If  the  question  is,  in  fact,  "can 
the  measures  detect  changes  in  increased  communications  (e.g.,  radio 
communications)?"  then  an  "incapacitation"  scenario  may  not  be  the  best 
alternative.  An  incapacitation  introduces  a  large  and  complex  set  of  task 
demands.  It  may  be  better,  in  fact,  to  introduce  a  specific  set  of  task  demands 
(e.g.,  radio  communications)  to  be  the  Captain's  assigned  task  in  addition  to  his 
normal  duties.  This  discussion  included  the  considerations  that  must  be  made  for 
all  phases  of  certification.  For  insterne,  if  an  incapacitation  scenario  is  used, 
where  does  incapacitation  of  the  first  officer  occur  (i.e.,  in  what  phase  of  flight), 
how  are  the  airline  procedural  differences  in  handling  an  incapacitation  accounted 
for,  and  when  incapacitated,  is  the  first  officer  removed  from  nis  seat?  Both 
issues,  incapacitation  and  communications  workload,  cover  workload  factors  of 
FAR  25,  Appendix  D,  which  were  not  manipulated  in  the  part-task  study.  The 
Boeing/Douglas  team  working  this  contract  is  evaluating  all  possibilities. 
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WRITTEN  COMMENTS  PROVIDED  BY  ATTENDEES 
WORKSHOP  #2 

FULL-MISSION  SIMULATION  SUGGESTIONS 


1 .  For  incapacitation  you  could  try  the  following: 


A.  Captain  says  Nl  have  briefed  flight  attendants,  the  F/O  is  being  cared  for 
by  a  physician,  ATC  has  been  advised,  EMS  is  standing  by,  you  will 
be  vectored  with  priority.  Also,  company  has  been  advised." 


B.  Have  Preston  suggest,  "You  look  tied,  why  don't  you  use  the  autopilot?" 
As  a  means  of  standardizing  autopilot  use. 


C.  Keep  your  high  workload  profile  as  it  is.  I  believe  it  will  measure 
sensitivity  at  high  levels. 


2.  Use  simple  scale  (say  Bedford  or  Airbus)  during  flight. 

3.  Do  TLX  post  flight. 

4.  Drop  SWAT  on  the  assumption  that  it  correlates  well  with  TLX. 


5.  Draw  the  pilots  from  a: 
-single  carrier 
•2  carriers  and  balanced 
-3  carriers  and  balanced. 


6.  Certification  would  take  the  autopilot  being  an  MEL  item  on  all  flights. 


7.  SWAT  did  well  on  high  workload,  so  do  a  regression  TLA-SWAT  check 
correlation.  Use  weights  for  the  next  simulation. 


8.  In  your  study,  you  used  visibility  and  wind  conditions  to  manipulate  workload 
levels.  However,  as  I  understand  it,  TLA  does  not  take  these  conditions  into 
account.  Consequently,  how  can  you  use  TLA  as  your  baseline  metric? 


9.  It  seems  that  a  lot  of  confounds  occurred  due  to  pilot  training  in  dealing  with 
anomalous  situations.  Some  pilots  dealt  with  emergencies  singlehandedly 
whereas  others  delegated  tasks  to  other  crewmembers.  Consequently,  you 
data  represented  different  types  as  well  as  levels  of  workload.  During  fall 


data  represented  different  types  as  well  as  levels  of  workload.  During  fall 
mission  simulation,  you  may  want  to  impose  stringent  operation  procedures  in 
order  to  control  for  some  of  these  confounds. 


10.  It  makes  sense  that  workload  should  be  consistent  as  possible  within  bounds 
of  "normal"  operations.  It  does  not  make  sense  to  try  to  combine  workload 
measures  for  pilots  who  use  autopilot  and  those  who  do  not. 
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1 1 .  Since  (a)  you  have  a  small  "N"  and  (b)  dispatch  with  autopilot  inoperable  is 
accepted  by  the  MEL,  why  not  use  a  ‘failed*  autopilot  in  your  tests. 

1 2.  i  recommend  that  your  experimental  procedures  be  reviewed  to  maximize  the 
degree  of  task  ana  task  load  commonality  within  the  bounds  of  "normal* 
operations. 

13.  If  a  secondary  task  is  used,  why  not  consider  realism  of  task  demand  as  one 
of  the  criteria?  This  may  not  be  feasible  for  available  secondary  task  list,  but 
the  concept  deserves  consideration. 

14.  You  have  a  two  crew-  three  crew  source  of  confusion  between  earlier  and 
present  test  plans.  Accordingly,  even  though  you  plan  emphasis  on  a  two 
crew  operation  in  coming  tests,  why  not  start  the  flight  with  three  than 
"disable"  one.  This  would  establish  a  connecting  link  between  the  two  series 
of  tests  as  a  meaningful  basis  for  comparison  across  tests. 

15.  Did  the  analytical  (TLA)  approach  include  the  variations  in  degraded  modes 
of  operation?  If  not,  a  higher  degree  of  correlation  might  be  found  between 
analytic  and  empirical  results. 

16.  Discussions  indicate  you  are  considering  changing  applications  of  tools  (e.g., 
SWAT  may  be  used  in  flight  when  it  was  earlier  used  post  flight).  When  the 
conditions  of  application  are  not  the  same,  there  is  a  nigh  likelihood  that 
subtle  differences  in  results  will  occur.  Context  dependency  cannot  be  ruled 
out,  as  thousands  of  graduate  students  found  out  in  studies  of  semantic 
differential.  Also,  there  is  a  strong  possibility  the  "inflight”  ratings,  will  be 
remembered  to  some  degree  ana  Rave  an  influence  on  "post  flight"  ratings. 

17.  lam  concerned  that  we  are  measuring  (or  attempting  to  measure)  pilots 
reaction  to  changes  that  affect  the  tasxs  he  must  perform.  We  choose  to  call 
this  reaction  "workload*.  The  experimental  design  you  have  chosen,  changes 
the  tasks  the  crew  must  perform,  as  a  funtion  of  tie  scenario  and  introduction 
of  malfunctions.  What  it  appears  you  wish  to  address  is  changes  in 
configuration  (from  a  base  line  aircraft  "A"  to  a  new  configuration  "B"),  and 
those  affects,  reaction  of  the  task,  workload.  Also  you  may  want  to  address 
changes  in  crew  size,  crew  training,  procedures,  environment,  etc. 

My  concern  is  the  level  of  reaction  that  we  measure  as  a  result  of  malfunctions 
introduction  into  a  mission,  may  have  very  little,  if  any,  correlation  to  other 
changes,  such  as  crew  station  configuration,  crew  size,  etc.  Research  must 
be  conducted  to  determine  if  the  measure  that  is  sensitive  to  one  change  is 
still  a  reliable  measure  for  other  changes  that  affect  crew  tasks  and  are  seen 
in  measurable,  significant  change  in  workload. 

18.  The  point  that  was  made  in  regard  to  the  coarseness  of  the  sensitivity  of  the 
parameters  to  detect  workload  is  well  taken.  Simply  because  the  parameters 
chosen  can  distinguish  increases  in  workload  during  anomalous  conditions,  it 
does  not  follow  that  they  can  detect  subtle  changes  in  workload.  Granted, 
one  must  investigate  the  capability  to  detect  coarse  events  initially,  but  you 
need  to  go  one  step  further  in  your  full  mission  simulation.  For  example, 
introducing  poorly  designed  display  formats  which  are  known  to  increase 
workload  might  be  an  approach  to  further  defining  the  sensitivity  level  of  the 
parameters  mat  have  been  chosen. 


USAF/FAA  REVIEW  OF  WORKLOAD  MEASUREMENT  METHODS: 
PART-TASK  SIMULATION  DATA  SUMMARY  WORKSHOP 


WORKSHOP  #2  PROCEEDINGS 
DISTRIBUTION  UST 


Donald  Armstrong 
Federal  Aviation  Administration 
4344  Donald  Douglas  Drive 
Long  Beach,  CA  90808 

Dr.  Randall  B.  Aust 
McDonnell  Douglas  Helicopter  Co. 
Building  531,  C240 
5000  East  Me  Dowell  Road 
Mesa,  AZ  85205 

Dr.  Frank  Austin 
FAA  ASF-4 

800  Independence  Avenue  SW 
Washington  DC  20591 

Dr.  R.  M.  Barnes 

CAA  House  Civil  Aviation  Authority 

45-49  Kingsway 

London,  WC2B  6TE  England 

Robert  P.  Bateman 
Boeing  Military  Aircraft  Co 
P.O.  Box  7730 
MS  K80-19 

Wichita,  KA  67277-7730 
Mr.  V.  Battiste 

NASA  Ames  Research  Center 
Attn:  MS-239-3 
Moffet  Field,  CA  94035 

Herb  Berry 
M.C.  75-11 

Douglas  Aircraft  Company 
3855  Lakewood  Blvd. 

Long  Beach,  CA  90846 

Dr.  Michael  A.  Biferno 
McDonnell  Douglas  Corporation 
E21  MC  209-01 
3855  Lakewood  Blvd. 

Long  Beach,  CA  90846-0001 


Dr.  AJvah  C.  Bittner 
Analytics 

2500  Maryland  Rd. 

Willow  Grove,  PA  19090 

Dr.  Richard  Blomberg 
Dunlap  and  Associate  East  Inc. 
17  Washington  Street 
Norwalk,  CT  06854 

Michael  R.  Bortolussi 
NASA  Ames  Research  Center 
Attn:  MS-239-3 
Moffet  Field,  CA  94035 

George  Boucek 

Boeing  Commercial  Airplane  Co. 
P.O.Box  3707 
MS  66-25 

Seattle,  WA  981 24-2207 

Sqn.  Ldr.  Harry  Britten-Austin 
AFWAL/FIGR 

Wright-Patterson  Air  Force  Base 
Dayton,  OH  45433 

Capt.  Robert  G.  Buley 
Check  Airman  -  Right  Training 
M/S  G-7200 
Northwest  Airlines  Inc. 

7500  Airline  Drive 
Minneapolis,  MN  55450 

Larry  C.  Butterbaugh 
AFWAL/FIGR 

Wright-Patterson  Air  Force  Base 
Dayton,  OH  45433 

Kim  Cardosi 

DOT/TSC 

DTS-45 

Kendall  Square 
Cambridge,  MA  02142 

Robert  E.  Chaney 

Boeing  Commercial  Airplane  Co. 

P.O.  Box  3707 

MS6L-38 

Seattle,  WA  98124-2207 


Rob  Chatfield 

Federal  Aviation  Administration 
Flight  Standards  Division,  ANM  270S 
17900  Pacific  Highway  S. 

C-68966 

Seattle,  WA  98168 

Dr.  Richard  E.  Christ 
ARI  Field  Unit 
P.O.  Box  6067 
Ft.  Bliss,  Texas  79906 

Philip  Condit 

Boeing  Commercial  Airplane  Co. 
P.O.  Box  3707 
MS  75-26 

Seattle,  WA  98124-2207 

Dr.  C.  W.  (Bill)  Connor 
Delta  Air  Lines,  Inc. 

1722  Ponce  De  Leon  Blvd. 

Coral  Gables,  Florida  33134 

Dr.  William  Corwin 
McDonnell  Douglas  Co. 

MC  209-01 

3588  Lakewood  Blvd. 

Long  Beach,  CA  90846-0001 

Dr.  William  L.  Derrick 
AF/DPXA 

The  Pentagon  Room  5C360 
Washington  D.C.  20330-5060 

Dr.  Alan  Diehl 

HQ  AF  16C/SER 

Norton  AFB,  CA  92409-7001 

Michael  Dresel 
Boeing  Military  Aircraft  Co. 

P.  O.  Box  7730 
MS  K78-08 

Wichita,  KA  67277-7730 

Dr.  F.  Thomas  Eggemeier 
Department  of  Psychology 


Chet  L  Ekstrand 
Boeing  Commercial  Airplane  Co. 
P.O.Box  3707 
MS  2T-62 

Seattle,  WA  981 24-2207 

Mr.  Terry  Emerson 
afwal/figr 

Wright-Patterson  Air  Force  Base 
Dayton,  OH  45433 

Kennith  Erdman 
FAA  ASF-4 

4344  Donald  Douglas  Dr. 

Long  Beach,  CA  90808 

Delmar  Fadden 

Boeing  Commercial  Airplane  Co. 
P.O.  Box  3707 
MS  01-81 

Seattle,  WA  98124-2207 

Dr.  A.  D.  Fisk 

160  Minehead  Road 

Irmo,  South  Carolina  29063 

Dr.  J.  P  Fouillot 
Laboratoire  de  Physiologie 
Faclute  de  Medicine  Cochin 
24  Rue  du  Faubourg  St.  Jacques 
Paris  75014  France 

Dr.  Richard  F.  Gabriel 
McDonell  Douglas  Co. 

Dept  E21  MC  209-01 
3855  Lakewood  Bh/d. 

Long  Beach,  CA  90846-0001 

Charles  W.  Geer 
Boeing  Aerospace  Company 
P.O.  §0X3999 
MS8K-03 

Seattle,  WA  98124-2499 

Daniel  Gopher 
Technion-IIT 

Faculty  of  Industrial  Eng.  &  Mgt. 
Haifa  32000  Israel 


Berk  Greene 

FAA  ANM-160S 

17900  Pacific  Highway  South 

C-68966 

Seattle,  WA  98168 
Peter  Hancock 

University  of  Southern  California 
Dept,  of  safety  Sciences 
Los  Angeles,  CA  90089-0021 

Dr.  Randell  Harris  Jr. 

NASA  Langley  Research  Center 
MS  152E 

Hampton,  VA  23665 

Wolf  Hebenstreit 
Boeing  Aerospace  Company 
P.O.  Box  3999 
MS  2K-58 

Seattle,  WA  98124-2499 

Capt.  Mel  Hoagland,  Jr. 

261  Columbine  Lane 
Evergreen,  CO  80439 

Jack  Howell 

934  SW  2nd  Street 

Boca  Raton,  FL  33432 

Dr.  Peter  V.  Hwoschinsky 
FAA  APM-430 

800  Independence  Avenue  SW 
Washington  D.C.  20591 

Dr.  Richard  S.  Jensen 
Ohio  State  University 
Dept,  of  Aviation 
Box  3022 

Columbus,  OH  43210 
Jon  Jonsson 

Douglas  Aircraft  Company 
Mail  Code  209-1 
3855  Lakewood  Blvd. 

Long  Beach,  CA  90846 


168 


t 


i 

l 

) 

i 


H.  H.  Knickerbocker 
M.C.  41-58 

McDonell  Douglas  Co. 

3855  Lakewood  Btvd. 

Long  Beach,  CA  90846 

Dr.  Arthur  F.  Kramer 
University  of  Illinois 
Psychology  Department 
603  East  Daniel  Street 
Champaign,  IL  61820 

Capt.  Thomas  J.  Kreamer 
1205  Croton  Drive 
Alexandria,  VA  22308 

Capt.  Roland  B.  Liddell 

P.O.  Box  584 

Corte  Maderra,  CA  94925 

Aileen  Logan 

Boeing  Commercial  Airplane  Co. 
P.O.  Box  3707 
MS  66-25 

Seattle,  WA  98124-2207 
Gary  Lopez 

Federal  Aviation  Administration 
New  England  Region,  ANE-265 
12  New  England  Executive  Park 
Burlington,  MA  01803 

George  Lyddane 
ANM-104N 
16005  Sabanahane 
Encino,  CA  91436 

Joseph  MacDonald 
Boeing  Commercial  Airplane  Co. 
P.O.  Box  3707 
MS  26-09 

Seattle,  WA  981 24-2207 

Teresa  L.  Mann 
Lockheed  Georgia  Co. 

Marietta,  Georgia  30063 


j 


Capt.  Alvah  S.  Mattox  Jr. 
12723  Brewster  Circle 
Woodbridge,  VA  22192 


100 


i 


! 

f 

t 

l 


| 


,s‘ 

» 


I 

i 


« 


V 


V 

j 

w 

V 

V 

> 


Lt.  Kim  Mazur 
AFWAL/FIGR 

Wright-Patterson  Air  Force  Base 
Dayton,  OH  45433 

Dr.  Thomas  M.  McCloy 
United  States  Air  Force  Academy 
Behavorial  Science  &  Leadership  Dept. 
Colorado  Springs,  CO  80840 

Dr.  James  McGuire 
M.C.  35-36 

McDonnel  Douglas  Co. 

3855  Lakewood  Blvd. 

Long  Beach,  CA  90846 

Dean  Melton 
FAA  ANMN-160S 
17900  Pacific  Highway  South 
C -68966 

Seattle,  WA  98168 

Dr.  Henry  W.  Mertens 

Human  Performance  Research  Unit 

FAA/CAMI 

Box  25082  AAC-1 18 

Oklahoma  City,  OK  73125 

Dr.  Sam  Metalis 
McDonnell  Douglas  Co. 

E21  MC  35-36 
3855  Lakewood  Blvd. 

Long  Beach,  CA  90846 

Thomas  R.  Metzler 
U.S.  Army  Research  Institute 
AVSCOM  PERI-IRA 
4300  Goodfellow  Boulevard 
St.  Louis,  MO  63120-1798 

Dr.  Neville  Moray 

University  of  Illinois 

Department  of  Industrial  Engineering 

Urbana,  1L  61801 

Daniel  Mottard 
DGAC-SFACT 
364  Rue  DuLouvre 
75001  Paris,  France 


■s 

*» 


Dr.  Susan  Norman 
Chief  Aeronautical  Human  Factors 
Research  Officer 
NASA-Ames  Research  Center 
Moffett  Field,  CA  94035 

Dr.  Robert  D.  O'Donnell 
NTI  Corporation 
4130  Linden  Drive,  Suite  235 
Claypool  Building 
Dayton,  OH  45432 

Capt.  J.  G.  Oliver  Jr. 

5097  Riverview  Rd.,  NW 
Atlanta,  GA  30327 

Harry  W.  Orlady 
Battelle  ASRS  Office 
625  Ellis  Street,  Suite  305 
Mountain  View,  CA  94043 

Dr.  Charles  Overbey 

Operational  Test  &  Evaluation  Division 

FAA  AST-200 

800  Independence  Ave.,  SW 
Washington  D.C.  20591 

Don  L.  Parks 

Boeing  Military  Airplane  Co. 

P.O.  Box  3707 
MS  33-22 

Seattle,  WA  981 24-2207 


Robt.  Paty 

Chairman  Safety  Committee 
APA 

P.O.  Box  5524 
Arlington,  TX  76005 

Thomas  Penland 
FAA  AFS-250 

800  Independence  Avenue  SW 
Washington  DC,  20591 

Dr.  Evan  Pickrel 
FAA  AAM-500 

800  Independence  Avenue  SW 
Washington,  DC  20591 


171 


James  Plakis 

FAA  New  England  Region 

New  York  Aircraft  Certification  Offce,  ANE  170 

181  S.  Franklin  Ave. 

Valley  Stream,  NY  1 1581 

Gary  B.  Reid 
USAF/AMRL 

Wright-Patterson  Air  Force  Base 
Dayton,  OH  45433 

George  Reid 

Boeing  Aerospace  Company 
P.O.Box  3999 
MS  8K-03 

Seattle,  WA  981 24-2499 

Dr.  John  M.  Reising 
AFWAL/FIGR 

Wright-Patterson  Air  Force  Base 
Dayton,  OH  45433 

Dr.  Alan  Roscoe 
Britannia  Airways  Ltd. 

Luton  Airport 

Bedfordshire  LU2  9ND  United  Kingdom 
Bill  Russell 

Air  Transport  Association 
1709  New  York  Avenue  NW 
Washington,  DC  20006 

John  Rye 

Civil  Aviation  Authority 
CAA  House 
45-59  Kingsway 

London  WC2B  GTE  United  Kingdom 

Diane  L.  Sandry-Garza 
Boeing  Commercial  Airplane  Co. 

P.O.  Box  3707 
MS  66-25 

Seattle,  WA  98124-2207 

Dr.  Richard  Schiffler 
ASD/ENEC 

Wright-Patterson  Air  Force  Base 
Dayton,  OH  45433 


Dr.  Samuel  B.  Schiflett 
USAF  SAM/VNB 
Crew  Technology  Division 
Brooks  AFB,  TX  78235 

Dr.  William  Shepherd 
FAA  AAM-550 

800  Independence  Avenue  SW 
Washington,  DC  20591 

Dr.  Clark  A.  Shingledecker 
NTI  Inc. 

Suite  235 

4130  Linden  Avenue 
Dayton,  OH  45432 

Robert  J.  Shivley 

NASA  Ames  Research  Center 

MS  239-3 

Moffet  Field,  CA  94035 

Eric  Shreder 
M.C.  35-65 

McDonnell  Douglas  Co. 

3855  Lakewood  Blvd. 

Long  Beach,  CA  90846 

Kim  Siler 

Lockheed-California  Co. 

P.O.  Box  551 
Burbank,  CA  91520 

Troy  D.  Sims 

FAA  -  Aircraft  Evaluation  Group 
601  East  12th  Street 
Kansas  City,  MO  64106 

Dr.  Don  Smith 
7310  Valburn  Drive 
Austin,  TX  78731 

Jean-Jacques  Speyer 
Airbus  Industrie 

Performance  &  Operations  Engin. 
Flight  Division  HQ  BP  No.  33 
31707  Blagnac  Cedex,  France 

Dr.  John  A.  Stern 
Professor  Psychology 
Washington  University  Box  1 125 
St.  Louis,  MO  63130 


Frank  Stogsdill 

Federal  Aviation  Administration 
601  E.  12th  Street 
Kansas  City,  MO  64106 

Harty  G.  Stoll 

Boeing  Commercial  Airplane  Co. 
P.O.  Box  3707 
MS  77-35 

Seattle,  WA  98124-2207 

Jerry  Stone 
M.C.  202-12 
McDonnell  Douglas  Co. 

3855  Lakewood  Blvd. 

Long  Beach,  CA  90846 

Preston  Suit 

Boeing  Commercial  Airplane  Co. 
Post  Office  Box  3707 
MS  2T-61 

Seattle,  WA  98124-2207 

Guice  Tinsley 
FAAAFS210 

890  Independence  AVE.,  SW 
Washington  D.C.  20591 

David  M.  Trousdale 
331  Cochran  Drive  NW 
Atlanta,  GA  30327 

Capt.  Frank  Tullo 
Program  Manager 
Cockpit  Resources  Management 
Continental  Airlines  Inc. 

7300  World  Way  West 
Los  Angeles,  CA  90009-2044 

Dr.  Joseph  P.  Tymczyszyn 
Boeing  Commercial  Airplane  Co. 
P.O.  Box  3707 
MS  66-25 

Seattle,  WA  98124-2207 

James  E.  Veitengruber 
Boeing  Commercial  Airplane  Co. 
P.O.  Box  3707 
MS  06-14 

Seattle,  WA  98124-2207 


174 


W.A.  Wainwright 
BAE  Civil  Division 
Hatfield 

Hertfordshire  ALIO  9TL  England 
Larry  Walrath 

McDonnell  Douglas  Company 
Building  107  MS  10 
P.O.  Box  516 
St.  Louis,  MO  63166 

Dr.  Christopher  D.  Wickens 
University  of  Illinois 
Department  of  Psychology 
603  East  Daniel  Street 
Champaign,  IL  61820 

Antoinette  R.  Williams 
NASA  Ames  Research  Center 
Bldg.  257-1 

Moffet  Field,  CA  94035 

Dr.  Ken  Williams 
McDonnell  Douglas  Co. 

E21  MC  35-36 
3855  Lakewood  Blvd. 

Long  Beach,  CA  90846 

Dr.  Glenn  F.  Wilson 
USAF 

AMRL,  AAMRL/HEG 
Wright-Patterson  Air  Force  Base 
Dayton,  OH  45433-6573 

Brien  Wygle 

Boeing  Commercial  Airplane  Co 
P.O.  Box  3707 
MS  24-10 

Seattle,  WA  98124-2207 

Angelos  Xidias 
FAAANM-111 

17900  Pacific  Highway  South 
C -68966 

Seattle,  WA  98168 


